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ABSTRACT 

3-D stereoscopic image recognition system based on fuzzy- 
neuralnetwork technology has been developed. The system consists 
of 3 parts; preprocessing part, feature extraction part, and 
matching part. Two CCD color camera images are fed to the] pre- 
processing part, where several operations including RGB hsv 
transformation are done. A multi-layer perceptron is used for the 
line detection in the feature extraction part. Then fuzzy match- 
ing technique is introduced in the matching part. The system is 
realized on SUN spark station and special image input hardware 
system. An experimental result on bottle images is also present- 
ed. 
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1 . Introduction 

The recent development of image processing and pattern 
recognition technology is remarkable. Many are put into the 
practical use in fields of industrial testing system, remote 
sensing and so on. It is difficult, however, to make a flexible 
vision system based on human experiences and human skilled knowi 
edge. On the other hand, fuzzy logic and neural network technolo- 
gy are applied over a lot of fields including the control and the 
image recognition, where a human like processing is introduced. 
By combining the both techniques, 2-D image recognition system 

has been realized and reported [1]. 

In this paper a newly developed image recognition system 
using the technique of the binocular vision with fuzzy neuralnet- 
work methodology is presented. The system is realized on SUN 
spark station and special image input hardware system with 2 cc 
color cameras. It consists of the following 3 parts; the pre- 
processing part where RGB-HSV transformation and other operations 
are done, the feature extraction part where a multi-layer percep- 
tron is used for the line detection, and the matching part where 
a fuzzy matching algorithm is introduced. Finally several experi- 
mental results on bottle images are presented in order to confirm 
the availability of the proposed system. 


2. Image recognition process 

Fig.l shows the outline of the presented image recognition 
process. It is roughly divided into 3 parts. 

In the preprocessing part binocular images of each 512 
pixels are taken by using 2 CCD color cameras. Several ordinary 
image processing operations and the concept of co or . 

are introduced in order to satisfy the quality requested in the 
feature extraction part. Then the contour features are extracted 
in the feature extraction part by using the multi layer type per 
ceptron and the factorization technique based on fuzzy logic. 
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Finally a 
part, where 


fuzzy matching algorithm is 
the result is presented in 


introduced in the matching 
terms of fuzzy set. 


Original Image 


Preproce 
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ssing Part 

cy by Color Feature 



Feature Extraction Part 

• Extracted Feature by Neuralnetwork 

• Segment Simplification 

• Segment Separation 

Standard Image 



Matching Part 
■ Fuzzy Matching 


Fig.l The outline of image recognition process 

S.Coiored region extraction based on color information 

thP rcr rwI!S Ut c lma9e fr ? m the camera is expressed by combining 
the RGB (Red, Green, and Blue) density. In the case of humanbe- 

° r lnformation is transmitted and is qualitatively recog- 

oi tL SyeS t0 3 lar9e brain (Perception center). And a lot 
of models are proposed to explain the process. Here HSV (Hue 
Saturation, and Value) hexagon cone color model [2] is used y 
introducing the RGB -HSV conversion. So the RGB color information 
j converted into three attributes of the hue, the satuJ^on 

ship func?!on; wh? r J three attributes are defined by the member! 
snip xunctions which are shown in Fig. 2. 

h u®anbeings extract the color features, the color 
lstnbution/tendency of the entire image is considered. For 

S1 L the image observed ^ composed of rather similar 
colors, then the color range to be recognized is set to be nar- 
rowed Such characteristics are expressed by fuzzy rules. An 

°5 f ^ Z2y rules is shown below. By introducing the fuzzy 
matching technique, the feature colors are extracted. 

[one example of fuzzy rules of feature color extraction] 

IF the hue of the object is closely distributed 

THEN the membership function of the hue should be narrowed. 


4. Line segmentation using multi-layer perceptron 

An image based on color information is converted to an line 
drawing image by using ordinary image processing technique. A 
multi layer perceptron is applied to scanning the line drawing 
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Fig. 2 Membership functions of three color attributes 

image and extract the directionality of lines. Here it should be 
noted that the non- learning data is generalized by the learning 
data in the Back Propagation Method of the perceptron model 

[3] [4] . _ . 

The input layer in the multi-layer perceptron corresponds to 

a part of the image. The teaching pattern in the learning process 
is a set of the typical lines of the input pattern, and the 
output pattern is the direction representing code called a chain 

code . . , 

The outline of the multi-layer perceptron is summarized as 

follows: The output of the i-th neuron in the first layer is 



u ! = f(A) , 

(i) 

where 

^ S ^ 1 -f exp ( — sj 9 

(2) 


Ni- 1 



- e - «i • 

(3) 


j=i 


The input of the (i,j) coordinate in the input frame is 


u\ = x{i%W,ilW) i ( 4 ) 

where % and / stand for the remainder and the quotient of divi- 
sion, respectively. The evaluation is given by 


E = \W~y^ 
l .=) 


( 5 ) 


where y x stands for the value of the i-th neuron in the teaching 

pattern^ input frame has a variable ratio which is determined by 
the ratio of the dark pixels to the bright pixels in the input 
frame. The position of the input frame is slightly changed in 
order to adjust the position of the center of gravity of the dark 
pixels to the middle of the input frame. The coordinate (G ? , Gy) 
of the center of the gravity of the dark pixels in the input 
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frame, and the ratio S of the dark pixels to the bright pixels is 
calculated as follows; H w p s 1S 


G* = 


j=i .=1 


G y = 


S = 


"•£*(>,;) 

1 = 1 
W H 

E • j 

■=i;=i 

1=1 
H W 

H • W 


( 6 ) 


( 8 ) 


Based on these input values the output of the multi-layer percep- 
tron is calculated. By moving the position of the input frame 
taking the output value into the consideration the line segment 
is traced. In the branching point of the line segment the input 
frame of the multi-layer percept ron also makes a branch and the 
line segment is traced in parallel. 


5. The simplification of the extracted line segment data series 

The output data series obtained by the multi-layer percep- 
tion represents the directionality of the line segment. It is 
classified into a group of straight line, curved line, and corner 
£>y using the membership function shown in Fig. 3. 

The simplified data represent the geometrical feature of the 
nput image. They can be understood in many ways with membership 



Fig. 3 Membership function of geometrical features 
6. Binocular stereoscopic vision 

6.1 Correspondence between left and right images 

Three dimensional binocular stereoscopic vision is realized 
by using in principle the difference between left and right 
images. But it is not so easy to make a correspondence of charac- 
teristic points between two images. 

In this study the correspondence is made by using the sim- 
plified data series of both images mentioned in section 5. The 
line segment correspondence can be made based on the distance 
^ e ^ ween line segments [5] . The both images are divided into sever- 
al line segment blocks and the similarity between the blocks are 
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line segments. Such a 


calculated which generates sub blocks of 
procedure continues and finally the correspondence of line seg- 
ments are obtained, where fuzzy logic is introduced especially 
the representation of the shape of line segment (c.f. Fig. Jj 
correspondence operation. 

6.2 Separation of objects . . 

The distance between the camera and the object is . 

based on the information of line segment correspondence obtained 
in 6.1, where the method in projective geometry [6] are intro 
duced. The position [a] in the 3D space and its corresponding 
position [b] in the image is connected by a translation matrix M 


and 


calculated 


The matrix M can be calculated from the data of 6 points. Then 
the both images are transformed into 3D space by applying this 
translation* matrix M. By doing a clustering procedure in 3D space 
the contour line of each object is extracted. 



Calculation of M | 


l 

Translation into 3D space < 


r 


Clustering 


U 

Contour line separation 


Fig. 4 Contour line separation in 3D space 


Humanbeings can recognize the target object to some 
even if some part of it is hidden. Such functions are realized 
here by introducing fuzzy matching technique. The recognition 
result here is the similarity between the extracted information 
mentioned in the section 6 and the standardpattern 1 ^° rfflatlon of 

The standard pattern information consists of t YV 

line, the coordinates of starting point and end point, the 
in the case of straight line, the curvature and the angle in the 
fAQ** of curved line and corner, and so on. 

Firstly the segment with the minimum y coordinate ^ found 
and is checked if it is the top part of the object by observing 
the left and the right segments. Then the data series are divided 
into two parts. The similarity of the segment dataagainstall 
standard pattern information is calculated. Then the relation 
between the segment data and the standard data with the maximum 
Smn^itTis checked if there exist contradictions Mw = ^‘der- 
ing other relations. By repeating this kind of P ro ^®“^ r ® \ 

final result is obtained. Table 1 shows the list of comparative 
features in the similarity calculation. Fig. 5 shows their mem r 
ship functions. 
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Table 1 A list of comparative features 

in the similarity calculation 


Attribute 

Comparative Feature 


• Start-End Coordinates 

Ap 

Straight 

• Length 

Al 


• Inclination 

AB 


• Start-End Coordinates 

Ap 

Curve 

• Chord Length 

Al 


• Angle at the Circumference 

AB 

Angle 

• Angle 

ab 




Fig. 5 Membership functions of comparative features 
8 Experimental result 

In order to confirm the availability of this method several 
experiments have been done, among which one result using bottles 
is shown. Observed image of bottles consist of straight lines, 
curved lines with various curvature, and corners. There exist so 
many similarly looking different bottles. So they are good for 
testing the presented method. 

Fig 6 shows several examples of original image observed by 
the CCD camera. Experiments were done for the single bottle, a 
pair of bottles, and three bottles. (The aim of latter two cases 
is to check the effect for occlusion. ) The result is summarized 
in Table 2, which shows the validity of the proposed method. 

9 .Conclusion 

Fundamental ideas and algorithms of 3D image recognition are 
proposed based on fuzzy neural network technique. A result of 
experiment on bottle images is also presented. The construction 
of real time 3D image recognition system for the purpose of robot 
vision is a part of future studies. 

This study was performed through Special Coordination Funds 
of the Science and Technology Agency of the Japanese Government. 
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Fig. 6 Sample patterns 





Fig. 7 Examples of experimental data 
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Table 2 Recognition results 

Category 
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